System and method for neural network based feature extraction for acoustic model development

ABSTRACT

A system and method are presented for neural network based feature extraction for acoustic model development. A neural network may be used to extract acoustic features from raw MFCCs or the spectrum, which are then used for training acoustic models for speech recognition systems. Feature extraction may be performed by optimizing a cost function used in linear discriminant analysis. General non-linear functions generated by the neural network are used for feature extraction. The transformation may be performed using a cost function from linear discriminant analysis methods which perform linear operations on the MFCCs and generate lower dimensional features for speech recognition. The extracted acoustic features may then be used for training acoustic models for speech recognition systems.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/985,560, filed on Dec. 31, 2015, the content of which is incorporatedherein by reference.

BACKGROUND

The present invention generally relates to telecommunications systemsand methods, as well as automatic speech recognition systems. Moreparticularly, the present invention pertains to the development ofacoustic models used in automatic speech recognition systems.

SUMMARY

A system and method are presented for neural network based featureextraction for acoustic model development. A neural network may be usedto extract acoustic features from raw MFCCs or the spectrum, which arethen used for training acoustic models for speech recognition systems.Feature extraction may be performed by optimizing a cost function usedin linear discriminant analysis. General non-linear functions generatedby the neural network are used for feature extraction. Thetransformation may be performed using a cost function from lineardiscriminant analysis methods which perform linear operations on theMFCCs and generate lower dimensional features for speech recognition.The extracted acoustic features may then be used for training acousticmodels for speech recognition systems.

In one embodiment, a method is presented for training acoustic models inspeech recognition systems, wherein the speech recognition systemcomprises a neural network, the method comprising the steps of:extracting acoustic features from a speech signal using the neuralnetwork; and processing the acoustic features into an acoustic model bythe speech recognition system.

In another embodiment, a method is presented for training acousticmodels in speech recognition systems, wherein the speech recognitionsystem comprises a neural network, the method comprising the steps of:extracting trainable features from an incoming audio signal using theneural network; and processing the trainable features into an acousticmodel by the speech recognition system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating an embodiment of an overall processof feature extraction.

FIG. 2 is a diagram illustrating an embodiment of a two layer neuralnetwork and its cost determination.

FIG. 3 is a diagram illustrating an embodiment of a recurrent neuralnetwork

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of theinvention, reference will now be made to the embodiment illustrated inthe drawings and specific language will be used to describe the same. Itwill nevertheless be understood that no limitation of the scope of theinvention is thereby intended. Any alterations and further modificationsin the described embodiments, and any further applications of theprinciples of the invention as described herein are contemplated aswould normally occur to one skilled in the art to which the inventionrelates.

Automatic speech recognition (ASR) systems generally use acousticfeatures as a front-end for recording the useful information in speechsignals. These acoustic features may commonly be Mel Frequency CepstralCoefficients (MFCCs) that are obtained by processing the incoming audiosignal using well-known signal processing techniques in the art. MFCCsare easily extractable from an audio signal and may be de-correlated,which avoids redundancy in the feature set. In general, ASR systems mayuse MFCCs and their derivatives for model development and recognition.However, MFCCs and their derivatives do not always differentiate betweensimilar sounds, such as similarly sounding audible phones, weakconsonants, weak fricatives, weak plosives, and slurred speech, to namea few non-limiting examples. Linear Discriminant Analysis (LDA) methodsthat are inspired by statistics have been used with limited success toaddress this drawback.

In an embodiment, non-linear transforms may be used to performoperations on the MFCCs and generate lower dimensional features, whichhave better discriminative properties than MFCCs for speech recognitionpurposes. The transforms may be generated by optimizing a separation ordiscrimination function using a neural network to non-linearly map theMFCCs for feature generation. Neural networks are known for excellentfunction approximation, such that neural networks are capable ofapproximating complicated functions between input and output withsufficient training data. In an embodiment, neural networks, areutilized to extract well-trainable features from the MFCCs as opposed tousing the neural networks at the probability calculation step of theprocess.

FIG. 1 is a flowchart illustrating an embodiment of an overall processof feature extraction, indicated generally at 100. The process offeature extraction illustrated in FIG. 1 uses a neural network and theextracted features may be processed into an acoustic model. The process100 may occur in a speech recognition system, specifically, to constructan acoustic model for use in an ASR system.

In operation 105, high dimensional MFCCs are input to a featureextractor. In an embodiment, input may also comprise a spectrum. In anembodiment, a spectrum comprises a well-sampled speech signal asobtained from a discrete Fourier transform. The features may be obtainedthrough processing an incoming audio signal using signal processingtechniques known in the art. Control is passed to operation 110 and theprocess 100 continues.

In operation 110, features are extracted from a neural network. Forexample, the high dimensional MFCCs undergo neural network featureextraction. Low dimensional neural network LDA features result from thefeature extraction. In an embodiment, neural network feature extractionoptimizes cost functions more effectively than LDA, generating neuralnetwork-LDA (NN-LDA) features. Activation function parameters may berequired in addition to weights and bias parameters. Parameters for theactivation functions may be learned during model training Activationfunctions may be linear, quadratic, or polynomial, and are capable ofhaving as many parameters as desired.

In an embodiment, the neural network utilizes pre-aligned feature dataand may be trained using a stochastic gradient descent method on thecost function. It should be noted that while LDA cost functions may beused for the stochastic gradient descent method, other cost functionsmay also be used. With LDA, for example, the cost function may representa quantity that broadly measures separation between different acousticclasses.

In an embodiment, referring to the LDA cost function as a non-limitingexample, the LDA cost function represents a measure of inter-classseparation or discrimination. The function may be mathematicallyrepresented by:Cost=−Trace(W ⁻¹ B)

where W represents the within class covariance matrix and B representsthe between class covariance matrix. The within class covariance matrixmay be defined as:

$W_{\alpha\;\beta} = {\sum\limits_{C = 0}^{N}{{P(C)}\left( \frac{1}{N(C)} \right){\sum\limits_{x \in C}{\left( {x_{i}^{\alpha} - {\mu(C)}^{\alpha}} \right)\left( {x_{i}^{\beta} - {\mu(C)}^{\beta}} \right)}}}}$

The mathematical formula for W_(αβ) represents the sum over thecovariance matrices of each class weighted by the fraction of times anindividual class appears in the overall classification, where P(C)represents the individual class, the α index extends over thedimensionality of a feature factor (1 to 195, for example), μ(C)represents the class mean, N represents the number of classes, and N(C)represents the size of the data set in each class. The between classmatrix may be defined using the N class means appropriately weighted as:B _(αβ)=Σ_(C=0) ^(N) P(C)(μ(C)^(α)−μ^(α))(μ(C)^(β)−μ^(β))

where μ represents the mean of all the class means, which may also bethe same as the global feature mean of the data set. In an embodiment, Bmay also be determined by subtracting W from the full covariance matrix.

The activation functions of the neural network may be modified to permitthe output features to be useful for training Gaussian Mixture Models(GMM) described below in operation 115. A non-linear function whoseoutput takes a range of values may be used instead of a tan h function.Polynomial functions with linear, quadratic, or higher degree may beused. The coefficients of these functions may be considered as trainingparameters for training in addition to the weights connecting two layersand the biases at each node neuron.

Training may be performed using a stochastic gradient descent methodusing the gradient of the LDA cost function with respect to the neuralnetwork weights. This gradient is expressed in terms of the W and Bmatrix gradients as:∂Cost=−Trace(W ⁻¹ ∂B)+Trace(W ⁻¹(∂W)W ⁻¹ B)

The cost gradient may be computed in a standard way usingbackpropagation of the layer derivatives by the chain rule. The matrixgradients are easily expressible in terms of the hidden layer and outputlayer gradients from the expressions for the within class covariancematrix and the between class covariance matrix. Gradients with respectto the function parameters are also included in the total gradient. Theinput to the network comprises class aligned MFCC data. Control ispassed to operation 115 and the process 100 continues.

In operation 115, a GMM is trained using low dimensional featuresobtained in operation 110 using well-known expectation maximization andmaximum likelihood methods. Control is passed to operation 120 and theprocess 100 continues.

In operation 120, an acoustic model is generated and the process 100ends.

In an embodiment, only the front-end of many ASR systems needs to bealtered without any additional changes in the back-end.

FIG. 2 is a diagram illustrating an embodiment of a two layer neuralnetwork and its cost determination, indicated generally at 200. Theweights may be shown as directed line segments connecting adjacentlayers. In FIG. 2, class aligned MFCC data is input 205 to the firstlayer 210. The class aligned MFCC data may be obtained from a previouslygenerated acoustic model. High dimensional input is preferable. Forexample, five (5) consecutive MFCC vectors and their derivatives, for atotal of 195, may be used. The output from the second layer 215comprises a lower dimensional (e.g., 40, for example) projection of theinput obtained by optimizing the LDA cost function 220 used to lowernetwork cost 225. The classes in the training data comprise thoseassociated with states in a GMM. The network weights and biases areinitialized with suitable values. Two useful initializations maycomprise: the unit matrix (a simple projection of the input to theoutput) and the LDA transform. The LDA transform may be obtained fromthe eigenvectors of the matrix W⁻¹B with only a subset of the largesteigenvalues being retained. Training may be continued over severalepochs with a stopping criterion to prevent very low costs. Theactivation function parameters may also be changed along with theweights and biases at every training epoch.

In an embodiment, the standard LDA may be extended with a wider contextfrom a recurrent network with linear activation functions and twolayers, to one input and one output layer. LDA transforms requireseveral frames of input to work to capture the contextual informationpresent on both sides of a speech utterance. Size may be reduced byusing a recurrent network with linear activation functions and trainingit with the LDA cost function. In an embodiment, the network trainedwith the LDA cost function is capable of generating features that may beused other than in acoustic model development.

FIG. 3 is a diagram illustrating an embodiment of a recurrent neuralnetwork, indicated generally at 300. The recurrent neural network forLDA has a wider context whose output at one instant 305 acts as input tothe output layer at the next instant 215. The effect of the training issimilar to that of using an infinite impulse response (IIR) filter withfewer taps instead of a large finite impulse response (FIR) filter.

The development of an acoustic model in operation 120 may be performedusing the neural network LDA features. The features are used to trainmodels using maximum likelihood and expectation maximization methods.For speech recording systems, this includes the usual steps of trainingmonophone models and triphone models using Gaussian mixtures. Modeltraining may be stopped when likelihood values stop rising and convergeto stable values. In an embodiment where the neural network features areweakly cross-correlated, the correlations may be ignored or removed byan additional linear transformation.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, the same is to be considered asillustrative and not restrictive in character, it being understood thatonly the preferred embodiment has been shown and described and that allequivalents, changes, and modifications that come within the spirit ofthe invention as described herein and/or by the following claims aredesired to be protected.

Hence, the proper scope of the present invention should be determinedonly by the broadest interpretation of the appended claims so as toencompass all such modifications as well as all relationships equivalentto those illustrated in the drawings and described in the specification.

The invention claimed is:
 1. A method for training acoustic models inspeech recognition systems, the method comprising: extracting highdimensional acoustic features from one or more input speech signals;extracting, using a neural network, linear discriminant analysisfeatures from the high dimensional acoustic features, the neural networkcomprising a plurality of neurons and a plurality of activationfunctions, wherein the neural network is trained by computing parametersof the activation functions and weights of connections between neuronsof the neural network using stochastic gradient descent on a costfunction based on training data; and training an acoustic model usingthe linear discriminant analysis features.
 2. The method of claim 1,wherein the activation functions comprise linear discriminant analysisfunctions.
 3. The method of claim 1, wherein the training data comprisescomprising prealigned feature data and linear discriminant analysisclass labels.
 4. The method of claim 3, wherein the cost functionrepresents a measure of inter-class separation between the class labels.5. The method of claim 1, wherein the cost function is a lineardiscriminant analysis cost function.
 6. The method of claim 1, whereinthe high dimensional acoustic features comprise mel-frequency cepstralcoefficients.
 7. The method of claim 1, wherein the linear discriminantanalysis features have a lower dimension than the high dimensionalacoustic features.
 8. The method of claim 1, wherein the neural networkis a recurrent network.
 9. The method of claim 1, wherein the trainingcomprises using at least one of maximum likelihood method and anexpectation maximization method.
 10. A system for training acousticmodels in speech recognition systems, the system comprising: aprocessor; and a memory in communication with the processor, the memorystoring instructions therein that, when executed by the processor, causethe processor to: extract high dimensional acoustic features from one ormore input speech signals; extract linear discriminant analysis featuresfrom the high dimensional acoustic features using a neural network, theneural network comprising a plurality of neurons and a plurality ofactivation functions, wherein the neural network is trained by computingparameters of the activation functions and weights of connectionsbetween neurons of the neural network using stochastic gradient descenton a cost function based on training data; and train an acoustic modelusing the linear discriminant analysis features.
 11. The system of claim10, wherein the activation functions comprise linear discriminantanalysis functions.
 12. The system of claim 10, wherein the trainingdata comprises prealigned feature data and linear discriminant analysisclass labels.
 13. The system of claim 12, wherein the cost functionrepresents a measure of inter-class separation between the class labels.14. The system of claim 10, wherein the cost function is a lineardiscriminant analysis cost function.
 15. The system of claim 10, whereinthe high dimensional acoustic features comprise mel-frequency cepstralcoefficients.
 16. The system of claim 10, wherein the lineardiscriminant analysis features have a lower dimension than the highdimensional acoustic features.
 17. The system of claim 10, wherein theneural network is a recurrent network.
 18. The system of claim 10,wherein the memory further stores instructions that, when executed bythe processor, cause the processor to train the acoustic model by usingat least one of a maximum likelihood method and an expectationmaximization method.